Determination of position from images and associated camera positions

ABSTRACT

The absolute position of a target object point is determined using a series of images of the scene with overlapping fields of view captured by a camera in positions arranged in at least two dimensions across the scene and position data representing the absolute positions of the camera on capture of the respective images. The images are analysed to identify sets of image points corresponding to common object points in the scene. A bundle adjustment is performed on the sets of image points that estimates parameters representing the positions of the object points relative to the positions of the camera associated with each image, but without using input orientation data representing the orientation of the camera. The absolute position of the target object point is derived on the basis of the results of the bundle adjustment and the absolute positions of the camera represented by the position data.

The present invention relates to determination of the absolute positionof target object points. This is often referred to as ‘pointmensuration’.

Point mensuration is required in wide range of military and non-militaryapplications, many of which require accurate determination of geographiccoordinates in a timely manner. Such applications include, but are notlimited to, indirect targeting of precision guided munitions, in whichcase extant rules of engagement adhered to by a military unit, derivedfrom both targeting policy and legal constraints, determine the requiredaccuracy, precision and timeliness.

In many applications, the object point can be observed but notapproached. In this case, point mensuration can be performed by takingand analysing an image of the object point captured by a camera andmeasurements in respect of the camera on capture of the image, includingthe absolute position of the camera and the orientation of the camera.By way of example, the camera and other sensors mounted on an aerialplatform, for example an unmanned aerial vehicle (UAV). Such UAVs areaircraft without a human onboard that may be autonomous or controlledremotely. UAVs are now widely deployed by military forces and are alsofinding uses in non-military applications. In the case of using anaerial platform, measurements of absolute position may be taken using aglobal positioning satellite (GPS) receiver, and measurements oforientation may be taken using an inertial measurement unit (IMU).

Some known approaches for point mensuration using images from a cameramounted on an aerial platform together with sensors for taking othermeasurements are as follows.

A first approach to point mensuration is to use measurements collectedby sensors on the platform, which represent the position and orientationof the platform, as well as the orientation and field-of-view of thecamera. On the basis of user input designating a target image point inthe image that corresponds to the target object point, trigonometry canbe used to calculate its geographic coordinates. For example, a chain ofgeometric transformations may be used to calculate the position of atarget using metadata specified in detail in Reference [1].Unfortunately this approach has several issues which renders itunsuitable for many point mensuration applications, as follows.

A first issue is that the metadata accuracy. The metadata provided bythe platform is frequently inaccurate, particularly regarding theplatform and/or sensor orientation. Due to the lever-arm effect, anysmall inaccuracy in orientation results in a large target positioninginaccuracy. Smaller platforms with relatively small and hence lessaccurate sensors tend to be operated relatively close to the target,while relatively large platforms with correspondingly large and hencemore accurate sensors tend to be operated significantly further from thetarget. For example, a 2° angular error at a slant range of 1000 myields the same target positioning error as a 0.1° angular error at aslant range of 20 km, namely 35 m (these figures are intended to beillustrative only, and not representative of any particular system).

In addition to estimating the geographic coordinates of a target, is itessential to know the uncertainty of that estimate. In order to do this,the uncertainties of each item of metadata must also be known. Theseuncertainties are rarely provided by platform and sensor manufacturers,although Reference [2] is seeking to address this.

A second issue is the metadata contents. Once the geometricalcalculations are performed, a hypothetical ray is ‘fired’ from thesensor into the scene. In order to determine the final position of thetarget, two approaches may be taken. In a first approach, the slantrange to the target may be measured, for example using a laserrangefinder. However, such a sensor is expensive and inconvenient, tothe extent that it is rarely available in smaller aerial platforms dueto the size, weight and power overheads. Furthermore, the risk ofdetection of the laser beam may limit covert use. A second approach isto estimate the slant range to the target by intersecting the ray with aDigital Elevation Model (DEM) of the scene. For adequately accurateresults, this requires an up-to-date, high-resolution DEM which may notbe available.

For the reasons detailed above, point mensuration is difficult toperform in this way.

A second approach to point mensuration is to use tie points. In thiscase, an image analyst manually defines a set of tie points between asingle frame of motion imagery and a controlled set of referenceorthoimagery, indicating where the same feature can be found in bothimages. Once enough tie points have been defined, the full geographicposition of the motion imagery frame may be established and pointmensuration performed. Two certified software tools for this process areCommon Geopositioning Services (CGS) and Precision Strike Suite SpecialOperations Force (PSS-SOF).

While this approach can derive highly accurate and precise coordinates,it requires the availability of both a suitably trained analyst as wellas suitable reference data. As such, this process is typically performedat a more strategic, rather than tactical, level and does not supporttime-sensitive targeting scenarios, such as troops in contact.

Accordingly, it would be desirable to provide determination of theabsolute position of a target object point that reduces or avoids someof these issues.

According to an aspect of the present invention, there is provided amethod of determining the absolute position of at least one targetobject point in a scene, the method using a series of images of thescene with overlapping fields of view captured by a camera in positionsarranged in at least two dimensions across the scene and position datarepresenting the absolute positions of the camera on capture of therespective images, the method comprising:

analysing the images to identify sets of image points in plural imagesof the series of images corresponding to common object points in thescene; and

performing a bundle adjustment on the identified sets of image pointsthat estimates parameters representing the positions of the objectpoints relative to the positions of the camera on capture of therespective images without using input orientation data representing theorientation of the camera on capture of the images, and deriving theabsolute position of the target object point on the basis of the resultsof the bundle adjustment referenced by the absolute positions of thecamera on capture of the respective images represented by the positiondata.

Accordingly, the method provides for accurate determination of theabsolute position of a target object point in a scene using a series ofimages of the scene with overlapping fields of view and position datarepresenting the absolute positions of the camera on capture of therespective images, for example GPS data acquired by a GPS receiver. Thisis achieved by making use of vision-based tracking technology thatanalyses the images to solve for the scene structure, that is thepositions of the object points, relative to the camera track, that isthe positions of the camera on capture of the respective images.

In particular, the method makes use of an image analysis that identifiessets of image points in plural images of the series of imagescorresponding to common object points in the scene. This image analysismay be performed using techniques that are known in themselves in thefield of machine vision. The image analysis has the effect of allowingthe method to track the common object points through the series ofimages as the basis for performing a bundle adjustment that again may beperformed using techniques that are known in themselves in the field ofmachine vision. Such a bundle adjustment analyses the images, inparticular the sets of corresponding image points, and estimatesparameters representing the positions of the object points relative tothe positions of the camera on capture of the respective images. Thismay take into account the absolute positions of the camera representedby the position data as a prior without deriving modified estimates ofthe positions of the camera.

The results of the bundle adjustment are referenced to the positions ofthe camera represented by the position data to provide an absoluteposition for a target object within the sequence of motion imagery.

The bundle adjustment intrinsically makes use of a relatively largeamount of information present in the information and therefore has thecapability of providing a high degree of accuracy and in particular agreater accuracy than the first approach discussed above of using adirect geometric transformation of sensor measurements which representthe position and orientation of a platform supporting a camera and theorientation and field-of-view of the camera.

As compared to the second approach discussed above, the ability toperform the method using a data processing technique relying on theimages and the position data avoids the need for a suitably trainedanalyst and suitable reference data, as well as allowing the method tobe performed much more quickly, effectively in real time for thepurposes of typical applications.

Furthermore, the method may be performed without reliance on inputorientation data measured by a sensor associated with the camerarepresenting the orientation of the camera on capture of the images toset initial estimates of the orientations of the camera on capture ofthe respective images. Thus it is not necessary to measure, andsubsequently store and transfer, the orientation, for example avoidingthe requirement for a suitable sensor such as an IMU associated with thecamera. Such orientation data is commonly acquired in association withposition data such as GPS data, and it might be envisaged that theorientation data would be desirable to provide accurate determination ofabsolute position since it would increase the amount of informationavailable to the bundle adjustment.

However, the present inventors have appreciated that accuratedetermination of absolute position can be achieved without inputorientation data in the case that the images are captured by a camera inpositions arranged in at least two dimensions across the scene, forexample positions along a curved path. In this case, the set of camerapositions as a whole effectively provides information on orientationbecause it extends in two dimensions across the scene. Conceptually,this may be considered as the set of positions of the camera being athree dimensional object the orientation of which is effectivelyrepresented by the absolute positions represented by the position data.Thus, sufficient information is provided to the bundle adjustment by theinput position data, without the need to additionally use inputorientation data representing the orientation of the camera on captureof the images.

According to a further aspect of the present invention, there isprovided an apparatus configured to perform a similar method.

To allow better understanding, an embodiment of the present inventionwill now be described by way of non-limitative example with reference tothe accompanying drawings, in which:

FIG. 1 is a schematic view of a system for determining the absoluteposition of a target object point in a scene;

FIG. 2 is a diagram of the components of the system;

FIG. 3 is a flow chart of a method of determining the absolute positionof a target object in a scene;

FIG. 4 is a flow chart of a step in the method of identifying sets ofcorresponding image points;

FIG. 5 is a set of four images highlighting unguided matches betweendescriptors;

FIG. 6 is a reference image of a scene showing camera positions andtracked features;

FIG. 7 is a flow chart of a modified method of determining the absoluteposition of a target object in a scene; and

FIG. 8 is a flow chart of a target tracking step of the method of FIG.7.

FIG. 1 illustrates a system for determining the absolute position of atarget object 1 in a scene 2 in the case that the scene 2 is on theground and the system is implemented using an aircraft 10 mounted with acamera 11 to capture aerial images of the scene 2. In this example, thesystem comprises the payload 12 of the aircraft 10 and a ground controlstation 13 which communicate with one another over a wireless datalink14, as illustrated in the diagram of FIG. 2.

The aircraft 10 may be a manned aircraft or an unmanned aircraft such asa UAV. In the case that the aircraft 10 is unmanned, an autopilot system25 is present to control the flight path of the aircraft 10. Theautopilot system 25 is controlled by the ground control station 13, inparticular by transmission of suitable control signals over the datalink14.

In one practical implementation, the aircraft 10 is a Blackstart UAVsupplied by Blue Bear Systems Research, which is a hand-launched UAV.The Blackstart UAV has a payload bay of dimensions 200 mm×130 mm thatholds the payload 12. In this implementation, the Blackstart UAV isprovided with a roll-axis gimbal in order to point the camera 11 at thecentre of an orbit of the UAV.

The payload 12 comprises the following components.

The payload 12 comprises the camera 11 that is mounted on the aircraft10 to capture images of the scene 2. In general the camera 11 may be anysuitable type of camera that is capable of capturing images ofsufficient resolution for the image processing described further indetail below. Many machine vision cameras would be suitable. In onepractical implementation, the camera 11 is a Prosilica GB2450C whichprovides uncompressed colour imagery of high resolution (2448×2050pixels) at a high frame rate (15 fps).

The camera 11 desirably has a relatively wild field-of-view to performthe camera tracking discussed in detail below. Combining the likelyslant range, platform stability parameters and the likely target sizeprovides for estimation of a range of suitable lenses 15. In one systemimplementation the lens 15 was a Computar M0814MP2.

The camera 11 supplies captured images over an ethernet connection 16 toa USB-ethernet adapter 17 that functions to repackage the images from anethernet format in to a USB2 format. The images are then transferredover a USB connection 18 and embedded capture system 19.

The payload 12 also includes a GPS receiver 20 which acquires GPSposition data and supplies this to the capture system 19. In general,the GPS receiver 20 may be of any suitable type. One possibility is forthe GPS receiver 20 to be part of the autopilot system 25 for theaircraft 10, if provided.

The capture system 19 is configured to perform encoding of the imagesand the position data for transmission over the datalink 14. The capturesystem 19 may be implemented by any suitable video processing system,for example a Computer-On-Module (COM).

One practical implementation of the capture system 19 is as follows.

In this implementation the capture system 19 may be a Computer-On-Module(COM), in particular a Gumstix Overo FireSTORM which is a COM basedaround the TI DM3730 System-on-Chip (SoC) which features ahigh-performance ARM Cortex-A8 general purpose processor, a specialisedTI C64x Digital Signal Processor (DSP) for high-performance, low-powercompute-intensive algorithms, a Video Accelerator for video encoding anddecoding, as well as a 3D Graphics Accelerator and numerous on-boardperipherals. The ARM Cortex-A8 general purpose processor on the TIDM3730 runs a conventional Linux operating system, allowing for rapidsoftware developments and debugging. The payload may operate in either amotion imagery or a still imagery capture mode.

The motion imagery capture mode uses the conventional H.264 compressionalgorithm, accelerated by the DSP, and transmits an MPEG-2 TransportStream over the User Data-gram Protocol (UDP) consisting of motionimagery and metadata including the position data.

The still imagery capture mode also transmits an MPEG-2 Transport Streamover UDP, but rather than motion imagery, a private data stream is usedto transmit high-quality JPEG-compressed still images. The DSP is usedto accelerate the compression algorithm.

The reason for the two modes is to better trade off resolution againstframe rate. In the motion imagery mode (at lower resolution but higherframe rate), the difference between successive frames is small, so theH.264 compression algorithm is most appropriate. In the still imagerymode (at higher resolution but lower frame rate), the difference betweensuccessive frames is large, and the motion estimation aspects of theH.264 compression algorithm struggle and produce poor results, thereforewe use the alternative JPEG compression algorithm. An alternative wouldbe to operate the H.264 encoder in an I-frame only mode, although the TIH.264 implementation is limited to a resolution of 1280×720.

Practically, images to be analysed are likely to be still images,although the motion imagery stream may be used to provide real-timeimagery for initial identification of a target object.

Due to the Bayer filter used to implement colour in the camera,resolution may be changed by windowing the sensor, rather than resizingfull images, which results in an effective reduction in thefield-of-view as the resolution is decreased.

Tight synchronisation between the motion imagery and metadata isdesirable. The use of the MPEG-2 Transport Stream provides this throughthe MPEG-2 Presentation Time Stamp (PTS), which allows the two streamsto be reference to the MPEG-2 Program Clock Reference (PCR).

The capture system 19 supplies the encoded images and position data to adatalink interface 21 configured to transmit the that encoded data overthe datalink 14. The datalink interface 21 contains suitable hardwareand software elements for providing data communications over thedatalink 14 in accordance with a communications protocol. Any suitablecommunications protocol may be used. In one practical embodiment thedatalink 14 is a Microcard SVIP 5.8 GHz datalink which provides abidirectional internet protocol communication using commercial WIFItechnology at up to 49 Mbps.

The ground control station 13 is configured as follows.

The ground control station 13 includes a datalink interface 22configured to receive data from the aircraft 10 over the datalink 14.The datalink interface 22 contains suitable hardware and softwareelements for providing data communications over the datalink 14 inaccordance with the communications protocol used.

The ground control station 13 further comprises an analysis apparatus 23that is supplied by the datalink interface 22 with the images andposition data generated by the payload 12 and communicated over thedatalink 14.

The analysis apparatus 23 stores the images and position data in amemory 24 and carries out image analysis discussed in detail below. Theanalysis apparatus 23 has a user interface 26 of any conventional type(e.g. keyboard etc.) for accepting user input. The analysis apparatus 23includes a display device 27 for displaying the images.

The analysis apparatus 23 may be implemented by a processor executing anappropriate program, although alternatively various functions such asimage processing functions may be implemented using appropriate hardwarecomponents. Most conveniently, the analysis apparatus 23 is implementedin a conventional computer apparatus such as a personal computer.

The system operates as follows.

The aircraft 10 is caused to fly over a scene 2 to be imaged. Once theaircraft 10 is overhead the scene 2, the aircraft 10, and hence thecamera 11, is controlled to fly along a flight path extending in atleast two dimensions across the scene.

In the simplest case, the flight path may be a curved flight path. Oneoption is for the flight path to be an orbit around a position on theground. In that case, that position may be designated by user input tothe ground control station 13 on the basis of images captured by thecamera 11 and transmitted to the ground control station 13. Thatposition may also be a target object position for the analysis describedin detail below.

In other cases, the flight path may comprise plural straight sectionsthat are offset from each other to provide the two-dimensional extent.

As the aircraft 10 flies along the flight path, the camera 11 iscontrolled to capture a series of images of the scene 2. As the flightpath extends in at least two dimensions across the scene, positions inwhich the images are captured are arranged in at least two dimensionsacross the scene.

These images each have a field of view 3 as shown schematically inFIG. 1. The fields of view 3 of the respective images overlap but arenot necessarily identical. Image capture is controlled to provide asufficient degree of overlap to perform the image analysis discussed indetail below. In many instances there will be mutual overlap between thefields of view 3 of every image, but this is not essential. One optionwhere the flight path is an orbit around a position on the ground is forthe camera 11 to be consistently aimed at that position so that eachimage has a field of view 3 centred on that position.

As the aircraft 10 flies along the flight path and the images arecaptured, the GPS received 20 outputs position data that represents theabsolute positions of the camera on capture of the respective images.

In the subsequent analysis, the position data needs to be synchronisedwith the timings of the image capture. This may be achieved in anysuitable manner. One possibility is for the synchronisation to beperformed by the capture system 19, for example by the capture system 19associating each captured image by the camera 11 with the position dataoutput by the GPS receiver 20 at the time of capture. Anotherpossibility is for the capture system 19 to transmit a continuous streamof position data output by the GPS receiver 20 including timeinformation and similarly to transmit the images also with timing data.In that case the analysis apparatus 23 may perform an initial processingstep to associate each image with the corresponding position data.

The ground control station 13 is supplied with the images and theposition data obtained in this way. The analysis apparatus 23 then usesthe supplied images 30 and the position data 31 to perform a method ofdetermining an absolute position of a target object point in the scenethat is illustrated in FIG. 3 and comprises the following steps.

In steps S1, the images 30 are pre-processed to provide them in asuitable format for the subsequent analysis. The pre-processing in stepS1 may include downsampling of the images and discard of colourinformation.

The colour information is not used by the feature matching framework,and full-resolution images take a long time to process. In addition tothis, the sensor pixel resolution often exceeds the spatial and temporalblur in the images, so not much useful information is lost bydownsampling, compared to the data already lost by imperfect focus andmotion blur. One simple downsampling strategy that may be employed is tobuild a Gaussian-filtered dyadic image pyramid, and discard the finestlevels of the pyramid until the remaining data is an image of less than2000 pixels width. This typically means that all of the images 30 willbe caused to be between 1000 and 2000 pixels wide, with the result thatimage features can be extracted in real-time.

In addition, the pre-processing in step S1 may include time decimationof frames, in the case that the supplied images 30 are a sequence ofmotion imagery rather than still images. A typical video camera mightrun at 15 or 30 frames per second, but many of the frames do not providemuch additional useful information (the same features will be trackedbetween many frames, but the baseline between adjacent frames will bevery small). Therefore most of the input frames may be discarded untilthe image rate is approximately 1-2 Hz. The exact decimation factor maycontrolled on the basis of user-input, or may be automatically computed,for example to provide a fixed number of frames in a single analysis, orso that subsequent camera positions are a minimum distance away fromeach other in space.

In step S2, the images 30 that have been pre-processed are analysed toidentify sets of image points in plural images of the series of imagescorresponding to common object points in the scene. Step S2 uses machinevision techniques which are known in themselves to identify the steps ofimage points that corresponds to one and other to serve as the basis fora bundle adjustment in step S3 described in detail below. Each setconsists of image points in plural images 30 of the series. Within asingle set, the image points may be in any number of the images 30, butideally all the images 30 of the series. Within each set, the imagepoints correspond to one another in that they each correspond to acommon object point in the scene 2.

In particular, step S2 consists of a series of steps shown in FIG. 4, asfollows.

In step S2-1 each image 30 is analysed to detect features within thatimage 30. In general, any type of image feature may be detected, usingany of a range of existing feature detectors known for image processing.Such features are conventionally also referred to as interest points orkey points. Typically, it might be desired to produce several hundredfeatures distributed as evenly as possible over the image 30. The exactnature of the features is selected to provide sufficient quantity, easyof acquisition and reliable localisation.

In one implementation, the detected features are corner features. Thistype of feature is suitable where the imagery contains sufficientstructure and texture to extract many such corner features stably fromeach image 30. In this context, a corner feature is a region of theimage with a predetermined degree of gradient change in 2 orthogonaldirections. A variety of corner feature detectors are known and may beused in step S2-1. An example of method of detecting corner featuresthat may be applied is disclosed in reference [3].

This technique ensures that in successive images, we tend to extractcorners corresponding to the same set of real-world 3D features.

Optionally, step S2-1 may be performed to detect features at pluralscale. This may be achieved by building a Gaussian-filtered dyadic imagepyramid (or to or where this is generated in step S1 to reuse it) andthen to perform the detection of features at each level of the pyramid.This allows image structure on each of the different scales tocontribute information.

In step S2-2, descriptors are generated in respect of each image 30 fromreceptive patches of the image 30 at the position of each featuredetecting in step S2-1. The purpose of the descriptors is to identifyfeatures that correspond across multiple images 30 in order to constructtracks. The descriptor therefore represents a small patch around afeature for use in a comparison in order to identify the correspondingdescriptors. Whereas the features are selected for localisation,descriptor is aimed at distinctiveness.

As to the patch of pixels around each detected feature used forgenerating the descriptors, the size is selected as a balance betweenthe discriminatory information contain that increases with size andperspective distortions between views that increase with size anddegrade the comparison of the descriptors. Due to noise and actualsimilarity of the input images, the descriptors can be arbitrarilyclose, despite stemming from an originally different point.

The descriptors may be generated in a range of different ways. Atsimplest, the descriptor could be the image data of the patch, but thatreduces the ability to detect matching descriptors in different images.Indeed, the problem with making direct comparisons (cross-correlation)of patches around features is that they appear different with changes ofintensity, colour balance, scale, and rotation. Accordingly, thedescriptors are preferably a transform of the patch.

Advantageously, the descriptors are chosen to be invariant to one ormore of scale, view angle (including rotation about the device's opticalaxis), intensity, colour and other potential variations due to theviewer and target locations, attitude, and the observer's equipment.

The descriptor may be any of a wide range of descriptors that are knownin themselves in the field of machine vision. One option is for thedescriptors to be so-called GLOH descriptors generated as described inReference [4] which display excellent viewpoint invariance anddiscriminability properties.

The descriptors generated in step S2-2 are compared to identifycorresponding descriptors in steps S2-3 and S2-4, as follows.

In step S2-3, there are detected sets of descriptors generated fromdifferent images that match each other. Matching may be determined usingany measure of proximity of the descriptors which are high-dimensionalvectors (say 128 elements), for example a distance function. A match isclassified where the measure of proximity indicates similarity above athreshold, that may be fixed or may be adaptively determined to obtain areasonable number of matches. For efficiency, anapproximate-nearest-neighbour data structure may be used.

One option is to use an approximate nearest neighbour algorithm based ona kd-tree to search for the best match to a given descriptor within aframe, as follows. The two (approximate) nearest neighbours to a featureare found by searching the tree, and two tests are applied to see if thecloser of the two neighbours is a good match. First, the squareddistance between the current feature and the nearest neighbour must bebelow 0.25, and second, the distance ratio of the nearest neighbour tothe second nearest neighbour must be less than 0.6. Typically, thisapproach might find several hundred unguided matches between a pair ofsubsequent frames.

The approach disclosed in Reference [5] may be used. Using theprinciples of hash-tables, proximity can be quickly established butmight yield false positives.

Step S2-3 is performed without reference to the position of the featuresto which the descriptors correspond, and so may be considered asputative. The aim is to detect pairs of descriptors in separate imageswhich are very similar. This indicates a high probability that the pairof descriptors come from corresponding locations in each image. However,matches are inevitably also found for pairs of patches which look alikebut which are not from the same part of the scene or which relate to amoving object, such as a car.

To avoid this, in step S2-4 there are selected matching descriptorsidentified in step S2-3 that also conform to a common mapping betweenthe positions in the respective images of the features corresponding tothose descriptors (i.e. guided matches). To do this, an assumption of anear-planar scene (where features on the ground should be on a singleplane, measured relative to the camera-to-scene distance) is used torobustly estimate a homography using the RANSAC algorithm disclosed inReference [6]. Matching descriptors that conform to this homography areselected, typically providing upwards of 100-200 unguided featurematches per frame.

However, step S2-4 is optional, because it has been discovered that fortypical scenes on the ground such matches do not provide much moreuseful information, compared to the unguided matches, but do serve toslow down the optimization procedure. It is believed that this is due tothe fact that guided matches are shorter-lived (and finer scale) thanmost unguided matches, and do not provide the extra constraints that alonger-lived track might.

By way of example, FIG. 5 shows sets of unguided matches across fourimages 30 actually captured. Adjacent images 30 are 8 frames apart in amotion image sequence, as even with this amount of time decimation,there is still plenty of overlap between images 30. Even though themotion is fairly jerky, the frame-to-frame motion is easily extractedfrom these feature tracks.

In respect of each set of corresponding descriptors derived in step S2-4(or step S2-3 if step S2-4 is omitted), the positions of the detectedfeatures from which the set of corresponding descriptors are generatedare identified as a set of image points corresponding to a common objectpoint and output from step S2. The term ‘track’ may be used to describethe set of image points. When a pair of image points are matched, it ischecked if the matched feature in the previous frame was already part ofa track. If so, the feature in the new frame is simply added to the endof this track. Otherwise, a new track consisting of the pair of featuresis created. With good orbit footage, it is possible to build long tracksover tens of frames. Longer tracks give the bundle adjustment describedbelow more information to constrain parameters, leading to faster (andmore precise) convergence.

In step S3, there is performed a bundle adjustment on the identifiedsets of image points output from step S2, i.e. the tracks. The bundleadjustment uses a non-linear optimisation technique. A wide range ofalgorithms known in themselves in the field of machine vision may beapplied. The preferred technique is to use a Levenberg-Marquardt bundleadjustment as disclosed in References [7] and [8].

The parameterization of a bundle-adjustment problem plays an importantrole in the stability and conditioning of the problem, as well as thespeed of convergence. Rotation parameters, in particular, requirecareful handling. In general, the parameters could include all of: thepositions of the object points (3D feature point positions); theorientations of the camera 11 in respect of each image 30 (3Dorientation); and the positions of the camera 11 in respect of eachimage 30 (3D camera position). The orientations of the camera 11 inrespect of each image may be represented using unit quaternions, Eulerangles or modified Rodrigues parameters for orientation. The latter hasadvantages described in Reference [9]. The key features are that noconstraints are needed (whereas quaternion representations requirere-normalization), and there are no singularities up to ±360° (whereasEuler angles suffer from gimbal lock).

In a preferred case, the bundle adjustment estimates parametersrepresenting the positions of the object points to which the imagepoints correspond, relative to the positions of the camera 11 on captureof the respective images 30 and also parameters representing theorientations of the camera 11 in respect of each image 30, but notparameters representing the positions of the camera 11 in respect ofeach image 30. Thus, the positions of the camera 11 in respect of eachimage 30 represented by the position data 30 are taken into account instep S4 as a prior, and modified estimates are not derived by the bundleadjustment. Thus, 3D absolute positions are retained to describepositions of object points. 3D positions are stored in a localEast-North-Up coordinate system, taking the first camera's latitude andlongitude, and an altitude of 0 m, as the origin.

Effectively this makes an assumption that the positions represented bythe position data 30 are exactly correct. Although this is clearly notthe case in fact, the nature of typical scenes 2 makes it difficult toimprove camera position estimates with the visual geometry any betterthan simple sample interpolation will provide. Camera orientations,however, are strongly constrained by the geometry of the scene 2. Byhalving the number of camera parameters for a given sequence, there isgained simultaneously a significant performance increase and improvementof the conditioning of the bundle adjustment problem, but without losingmuch precision.

In view of this, it is preferred that the position data is pre-processedand interpolated, rather than using some simpler technique such assample-and-hold, but this is trivial if accurate timestamps areavailable with the GPS position data (which is essentially the same asthe requirement for the images and position data to bewell-synchronized).

Regarding conditioning, to ensure that the implicit Levenberg-Marquardtspherical trust region assumption is valid, the residuals and parametersare conditioned to the unit sphere. To achieve this, the bundleadjustment uses stored data representing the intrinsic properties of thecamera. Such camera calibration will now be described.

Camera calibration provides the best possible target precision byremoving certain optical effects of the camera, and allowing modellingof the imaging process using a simple pinhole projection model. One aimsto achieve two objectives with camera calibration, namely correction ofradial lens distortion and estimation of the focal length of the lensrelative to sensor size

A pinhole camera model simply states that a 3D point in the camera'sframe of reference [X_(c) Y_(c) Z_(c)]^(T) will project to the imagepoint [x y]^(T) as follows (with Z_(c) in the direction normal to theimage plane, i.e. along the camera's look direction):

$\begin{bmatrix}x \\y\end{bmatrix} = \begin{bmatrix}{{f_{x}X_{c}\text{/}Z_{c}} + c_{x}} \\{{f_{y}Y_{c}\text{/}Z_{c}} + c_{y}}\end{bmatrix}$

where f_(x), f_(y) are focal length parameters in the X and Y directionswithin the image plane, and c_(x), c_(y) describe the principal point ofthe camera. For cameras with square pixels and a symmetrical lens,f_(x)=f_(y), and for cameras with a w×h pixel sensor centred on theorigin, c_(x)=w/2, c_(y)=h/2. In this case, image plane coordinates willmap to pixel coordinates.

This may be expressed in homogeneous coordinates as the following linearequation:

$x_{h} = {{KX}_{c} = {\begin{bmatrix}x_{h} \\y_{h} \\s\end{bmatrix} = {\begin{bmatrix}f_{x} & 0 & c_{x} \\0 & f_{y} & c_{y} \\0 & 0 & 1\end{bmatrix}\begin{bmatrix}X_{c} \\Y_{c} \\Z_{c}\end{bmatrix}}}}$

where x=x_(h)/s,y=y_(h)/s.

So far, four parameters need to be estimated (two focal lengthparameters and two principal point parameters). For many cameras, thismight be reduced to three parameters (when f_(x)=f_(y)) or even one(when the sensor lies on the lens centre line, so that c_(x)=w/2 andc_(y)=h/2). Real cameras include an imperfect lens, however, and thelens model that we shall use assumes that all lens distortion is purelyradial. More complex lens models, including skew and tangentialdistortion, are possible, but radial distortion tends to be the majorsource of error. One possible radial distortion equation is:

x _(d) =x _(u)(1+k ₁ r ² +k ₂ r ⁴)

r ² =x _(u) ^(T) x _(u)

where x_(u) indicates undistorted pre-projection image planecoordinates, and x_(d) indicates distorted image plane coordinates. Thisequation is therefore applied after dividing camera-space coordinates byZ_(c), but before projecting into pixel coordinates with f_(x), f_(y),c_(x) and c_(y). Therefore in this implementation, the stored datarepresents a total of six internal camera parameters (including k₁ andk₂) for any given camera 11 and lens 15.

Applying the camera calibration, the 2D lens-distorted image points thatmake up the tracks, are taken and then unprojected and undistorted, sothat they lie in the “ideal” image plane. Unprojection is the process oftranslating the image coordinates so that the principal point is at[00]T and then dividing by focal length so that we are in thelens-distorted image plane. The inverse of the lens radial distortionequation is applied so that the tracks conform to the simple pinholeprojection model. There is no direct analytic solution, so an iterativeapproach is used with 5 iterations of the equations:

r² = x_(k)^(T)x_(k)$x_{k + 1} = {\frac{x_{0}}{1 + {k_{1}r^{2}} + {k_{2}r^{4}}}\mspace{14mu} {with}}$x₀ = x_(d) x_(u) = x₅

The positions are conditioned by computing the mean and standarddeviation of all of the Cartesian East-North-Up positions of the objectpoints and cameras 11. This may not result in an exactly sphericalconditioning, especially for the first bundle-adjustment (as the initialimage points are only rough estimates), but it is close enough for theoptimizer to run without encountering numerical ill-conditioning or poortrust region validity. Without this conditioning, it is unlikely thatthe optimizer would find the correct minimum.

Methods of deriving the stored data representing the intrinsicproperties of the camera, and alternative methods for estimating suchproperties in the bundle adjustment are discussed below.

The bundle adjustment in step S3 is initialized as follows, that is toset initial values for all of the parameters in the problem.

The initialization uses an estimate of the ground plane elevation. Whileit might be expected that the initial ground plane estimate is fairlyimportant for the 3D point back-projection initialization step to yieldreasonable results, typical bundle adjustment techniques such as theLevenberg-Marquardt bundler can actually cope with very large errors ininitial locations without needing more than one or two extra iterations.Experiments providing a wide range of values of initial elevationindicate that having a ground plane that is merely in front of thecameras is good enough for fast convergence. Therefore, in the case thatthe flight path is curved, the present method may derive an estimate ofthe ground plane elevation from a geometrical measure of the curvatureof the path of the absolute positions of the camera 11 in respect ofeach image 30. A range of geometrical measures may be used. Suitablegeometrical measures might be: subtracting half the orbit radius fromthe lowest point in the flight path, and using this as the ground plane;or looking up the nearest valid value from a coarse Digital ElevationMap (DEM).

As an alternative, the method may estimate of the ground plane elevationto be a predetermined value (say 400 m) below the lowest point in theflight path.

Initial estimates of the camera orientation are set as follows, inparticular without using input orientation data representing theorientation of the camera on capture of the images. Since the system isoperating on images 30 captured from a camera 11 in an aircraft 10, itmay reasonably be assumed that the camera 11 will be pointing in thegeneral direction of the target for most of the flight. Initialestimates of the camera orientation are derived that assume each image30 is captured with the optical axis aligned with a common object pointin the scene 2, for example when the flight path is an orbit being inthe centre of the orbit.

The camera orientation is estimated from a simple geometricalcalculation, using the estimate of the ground plane elevation. That is,once we have a notional point in 3D space that the cameras 11 may belooking at has been determined, the orientation of each camera 11 isinitialised so that its optical axis passes through this point, and itsup-vector is in a vertical plane.

Where there is the possibility of the camera being mounted upside-down,the analysis apparatus 23 may accept user-input of a parameter thatindicates whether the camera is in fact mounted upside-down, as thebundle adjustment is unlikely to converge if the initial orientationerror is more than 90°, and choose an upwards or downwards up-vector asappropriate.

While computing orientations in this way is unlikely to be particularlyaccurate, all we need is for the true positions to be in front of thecamera 11, and for the cameras 11 to be less than about 90° of roll awayfrom the correct value. This technique works for all of the testsequences, and is likely to work for any other sequence thatapproximates an orbit, providing that the camera 11 does not perform afull roll during the sequence capture. Therefore, more generally, theinitial estimates of the camera orientation may take any values within90° of each other.

Initial estimates of the positions of the object points relative to thepositions of the camera 11 in respect of each image 30 are set asfollows. Based on the initial estimates of the orientations of thecamera 11 in respect of each image 30, a track that is observed bymultiple cameras 11 is taken, and the 3D geometry of the track isback-projected using linear or non-linear methods. This is possiblebecause there is a known baseline between the cameras 11, and as aresult, the point can be triangulated.

However, the camera orientation estimates are very approximate, and arelikely to have errors of several degrees, to the extent where thegeometry of the problem may indicate that a point is behind the cameraor near infinity. To alleviate this problem, it is assumed that there isa horizontal ground plane somewhere beneath the camera 11, and theinitial estimates of the positions of the object points are constrainedto lie on this plane, using the initial estimate of the ground planeelevation derived as discussed above. The resulting linear equations aresolved for minimum reprojection error using a Singular ValueDecomposition (SVD) for numerical stability. It should be noted thatonce the initialization step is complete, the concept of a ground planeis discarded and the positions of the object points are allowed to moveanywhere in world space during the optimization.

Where a Levenberg-Marquardt optimizer is used in the bundle-adjustment,the following process is performed. There is solved the linear system:

(H _(k) +λD)δx _(k) =−g _(k) ^(T)

in order to perform the update step:

x _(k+1) =x _(k) +δx _(k)

A sparse cameras-and-features system of equations is optimised, based ondivision of the modified Hessian H_(k)+λD into four quadrants:

${\begin{bmatrix}U & W \\W^{T} & V\end{bmatrix}\begin{bmatrix}x_{1} \\x_{2}\end{bmatrix}} = \begin{bmatrix}y_{1} \\y_{2}\end{bmatrix}$

Gaussian elimination is applied to this set of equations to solve thereduced top row:

(U−WV ⁻¹ W ^(T))x ₁ =y ₁ −WV ⁻¹ y ₂

via Cholesky factorization. The top row is much quicker to solve,because the number of cameras (hence the number of blocks in U) is muchsmaller than the number of features (and blocks of V, block columns ofW). However, some substantial sparse matrix multiplications need to beperformed before we can run the Cholesky factorization (the inverseoperations are cheap because U and V are block-diagonal).

The sparse BLAS library functions implemented in the Intel Math KernelLibrary have been used to improve the speed of the sparse matrixoperations at the core of the bundler. Specifically, the U and Vmatrices are now stored in the Block compressed Sparse Row (BSR) format,and the W matrix is stored in the Compressed Sparse Row (CSR) format.However, one of the key functions, that is the matrix-matrixmultiplication of V⁻¹ and W^(T), cannot be achieved with the availableSparse BLAS routines without first converting WT into a dense format.Even better performance may still be achievable by storing W in arectangular BSR format (as disclosed in Reference [10]) and hand-codinga square BSR/rectangular BSR matrix-multiplication algorithm (ideallyusing hardware-optimized vector instructions).

One possibility for initializing and updating the Marquardt parameter λ,is to initializing to 10⁻⁵ and then dividing by 10 on a successfulupdate, or multiplying by 10 if the objective function increases. Whilethis strategy does generally find the correct solution, it tends to havepoor convergence behaviour, as it can take several iterations to reachthe appropriate value of λ for the current position in the objectivefunction parameter space, noting that λ controls the extent to which theLevenberg-Marquardt optimizer behaves like a gradient descent optimizer(large λ) and a Gauss-Newton optimizer (small λ), by defining the trustregion size. In general, the bundle-adjustment problem is well behaved,even in the initial iterations, so we may use a small value of λ tostart with.

Another possibility that is preferred is to use the method outlined inReference [11] to control the parameter step updates, essentially usingthe gain factor to decide how well-behaved our function is, and weensure that our initial value of λ is normalized to the size of themaximum diagonal element in the information matrix. The net result ofthis change is that faster convergence is observed, due to feweriterations of the optimizer being required to obtain a given degree ofprecision.

Bad track removal will now be considered. It is possible that despitethe robust homography estimation employed in the feature matcher, sometracks contain bad matches. Such tracks will have a dramatic effect onthe solution calculated by the bundler, so we continue to use arobustification strategy which is to delete tracks with largereprojection error after every n bundler iterations, where n takes asuitable value such as 7 to avoid unnecessarily deleting good trackswhile the bundler is still converging.

In addition to this, the maximum number of tracks that may be deleted inany one iteration can be limited (currently to 10 tracks), so that ifthe bundler has converged on a poor solution due to one or two badtracks, not so many good tracks are deleted as to prevent convergence ona good solution in future iterations. The tracks are sorted such thatthose with the worst reprojection errors get deleted first.

In summary, the bundle adjustment comprises the following processes:

-   -   Initialize camera orientations    -   Initialize feature positions by back-projection on an estimated        ground plane    -   Refine these parameters by minimizing track reprojection error        with a Levenberg-Marquardt bundler    -   During the bundle-adjustment, remove tracks with poor        reprojection error after a certain number of iterations

The bundle adjustment comprising these processes is repeated every timea predetermined number of images 30 is received, for example every 40images 30.

By way of example, FIGS. 6( a) to (d) show the same reference image(taken from Google Earth in this instance) of a scene 2 being imaged bythe system which is overlaid with marks indicating the positions of thecamera 11 (green lines) on capture of successive images 30 and theposition of detected features (red dots) that correspond in the capturedimages 30. FIGS. 6( a) to (d) show the progression of camera positionand position parameters in the bundle adjustment problem as moresuccessive groups of 40 images 30 are added to the solver. It can beseen that as more images 30 are used, the estimated positions (red dots)become better aligned with the reference imagery used for visualization.The fine adjustments to camera orientations cannot be seen at thisscale, but the overall orbit shape shows why the initialization strategy(cameras pointing inwards) is valid.

Bounding problems may be considered given the use of many cameras. Whilethe improved Levenberg-Marquardt solver code allows one to solve fairlylarge bundle adjustment problems (≈100 cameras) in a few seconds, thetotal bundle size needs to be limited to avoid slow performance and highmemory usage on extremely large problems. As a result, when there aremore than a predetermined number of images 30, for example 200 images30, the method ceases adjustment based on the oldest images 30 in theset (since these will have the least influence from the newestfeatures). In practice, this means that bundle adjustment always runswithin a few seconds, and long sequences may be used. However, littleprecision benefit is gained from more than one full orbit of a target,or using more images per orbit, so this limit is rarely reached inpractice.

When all the images 30 have been processed, the results of the bundleadjustment are the estimated parameters, which include the parametersrepresenting the positions of the object points corresponding to thesets of image points relative to the positions of the camera 11 oncapture of the respective images 30. Since the bundle adjustment isreferenced by the absolute positions of the camera 11 on capture of therespective images 30 represented by the position data 31, theseparameters also represent the absolute positions of the object points

In step S4, the absolute position of any number of target object pointsis derived on the basis of these results, as follows. The analysisapparatus 23 receives user input 32 designating one or more target imagepoints in at least one of the images 30. This may be done for exampleusing a cursor as part of the user interface 26. The target image pointcan be the position on the ground designated as described above tocontrol the flight path, or can be a fresh position arbitrarily selectedby the user from the images 30. The target image point corresponds tothe a target object point in the scene 2.

In step S4, a group of image points located around each designatedtarget image point are selected. Each target image point corresponds tothe a target object point in the scene 2. Accordingly, in step S4, theposition of each target image point is interpolated from the absolutepositions of the selected group of object points that have beenestimated in step S3.

This method provides for accurate determination of the absolute positionof the target object point without needing to use orientation datarepresenting the orientation of the camera on capture of the images,thereby avoiding the need for a suitable sensor such as an IMUassociated with the camera 11 on the aircraft 10 and avoiding the needto transfer such data from the aircraft 10 to the ground control station13. All that is needed are the overlapping images 30 with the associatedposition data 31.

There will now be discussed the camera calibration used to determine thestored data representing the intrinsic properties of the camera 11.

There may be applied the usual approach to camera calibration, that isto capture several images of a known 3D scene with the camera 11, findsalient points from the 3D scene within the images, and then compute thecamera model parameters by comparing the observed image locations to theexpected locations from a simple pinhole camera model. To automate theextraction of salient points in the image, a checkerboard pattern iscommonly used, where corners of the grid squares are easy to detect.Since the size of the grid squares is known, the 3D scene is known up toa rotation and translation (the checkerboard may be translated along 3axes relative to the camera, and rotated about 3 axes as well, giving 6degrees of freedom). As there are many more grid corners than parameters(camera optical parameters plus the pose degrees of freedom), one maysolve for the best set of parameters that minimize the reprojectionerror. A Levenberg-Marquardt optimizer is typically used for this step.

To get the best estimate of the camera's optical parameters, especiallythe radial distortion parameters, it is important to ensure that thecheckerboard covers a large proportion of the image. This has the effectof covering areas where radial distortion is largest (near the edges ofthe images), as well as making the checkerboard corner location stepmore robust to image noise and the aliasing process inherent incapturing an image as an array of pixels.

Both free and commercial software packages are available to performcheckerboard calibration (e.g. the OpenCV library'sfindChessboardCorners and calibrateCamera functions, and the MATLABCamera Calibration Toolkit).

Although in theory, a fixed-lens, fixed-focus camera only needs to becalibrated once, this is not the case in practice. Thermal expansion ofcomponents, light knocks, vibration, and other physical forces acting onthe lens and camera components, can change the relative positions of thelens and sensor. For the present method, image errors of a few pixelscan introduce geographical errors of a few metres. For best results,therefore, a camera in frequent use should be re-calibrated on a regularbasis. If the camera 11 is mounted underneath a small UAV that receivesknocks upon landing, re-calibration before each flight might benecessary.

A camera 11 forming part of a UAV sensor payload will typically beimaging objects that are many hundreds of metres away, and as such, thelens 15 on such the camera 11 will typically be set to focus at a largedistance (perhaps infinity). However, when calibrating with acheckerboard, it is required that the checkerboard covers a largeproportion of the image. If a small checkerboard is held close to thecamera, 11 it will fill the field of view 3, but the image will be badlyout of focus. Ideally, an extremely large checkerboard would be held along way from the camera (maybe hundreds of metres away), but such acheckerboard would be very large (say the size of a house)

One solution is to build and use such a large checkerboard with theobvious disadvantage that the aircraft 30 must always fly over it beforeit may be used elsewhere, and if the camera 11 needs to be re-calibratedfrequently and in different locations, this might not be practical.

Another solution is for an expert user to manually find correspondencesbetween features in an aerial image taken by the camera 11, and acontrolled georeferenced image that has been previously generated viasome other means. These correspondences are known as tie points, butsuffer from the disadvantages that reference imagery may be unavailableand that the manual process is slow.

Another solution is to calibrate with a smaller checkerboard, say of theorder of 1 metre square. This is difficult as it will be badly out offocus (if it fills the image), but solutions are known, for exampletaking repeated images with the checkerboard in different imagelocations to build up coverage of the field of view, or using a cornerrefinement algorithm to localise a corner that is subject toconsiderable blurring to a reasonable degree of accuracy. Alternatively,the aperture of the lens may be decreased, which increases thedepth-of-field without changing the lens calibration parameters. Byfinding a compromise between these factors, a reasonable calibration maybe achieved with a checkerboard small enough to carry by hand.

The above techniques are suitable for cameras 11 with a lens 15providing a fixed field-of-view and a fixed focus lens. Where the lens15 is a zoom lens, the zoom position must also be taken into account,for example on the basis of metadata representing the zoom state at thetime of image capture.

As an alternative, the bundle adjustment may estimates parametersrepresenting the intrinsic properties of the camera. This complicatesthe bundle adjustment but is feasible using known machine visiontechniques, for example as are commonly used by the entertainmentindustry to compute camera tracks from motion imagery. Whilst thisalternative might be less accurate, it opens up possibilities forperforming the method on data collected from a much wider variety ofsources. Various modifications to the method described above arepossible, some examples of which are as follows.

A first modification is to perform a target tracking approach bymodifying the method as shown in FIG. 7 to replace step S4 by a step S5performed prior to the bundle adjustment of step S3 (step S5 being shownin FIG. 7 as occurring after step S2 but optionally being performedbefore that).

Step S5 uses the user input 32 designating one or more target imagepoints in at least one of the images 30, as described above. In step S5,in respect of each designated target image point, there is identified aset of target image points in plural images 30 of the series of images30 that correspond to that designated target image point and hence tothe target object point. This is done using a machine vision techniqueas follows.

Target tracking has one important limitation, namely that the targetdoes not change. Meeting this requirement is dependent on the userselecting an appropriate target. Clearly it is impossible to assign asingle position to a moving target, but there is an additionalconstraint that the target does not continuously change shape (e.g. anobject that is rotating, or perhaps deforming in the wind), because suchtargets are difficult to reliably track in motion imagery.

The target tracking also assumes that image-to-image perspective changesare small, such that the changing appearance of the object in subsequentframes may be modelled by an affine transform. This is a reasonableassumption given that the camera 11 is expected to fly overhead thesubject, with a reasonably small baseline between images 30 compared tothe overall size of the flight path.

A further assumption used is that the scene 2 in any given image 30 isapproximately planar. This corresponds to an assumption that the changein height of structures within the scene 2 is small compared to thedistance from the scene to the camera-again, a very reasonableassumption in the usual case of a aircraft 10 flying over a scene 2 at amoderate distance.

Step S5 comprises steps S5-1 to S5-3 shown in FIG. 8 and performed asfollows.

In step S5-1, there is estimated a homography between the one of theimages 30 in which the designated target image point is designated andother images 30. Based on an assumption that the scene 2 isapproximately planar compared to the scene-to-camera distance, adjacentimages 30 are related by with a homography H. This is a 3×3 matrix thatrelates (homogeneous) image coordinates between images A and B suchthat:

x _(A) =Hx _(B)

In step S5-1, this homography is estimated using the RANSAC algorithmdisclosed in Reference [6].

In step S5-2, the position of the designated target image point in eachof the other images 30 is predicted using the homography.

As the predicted position is unlikely to be exactly correct, in step S5,a matching approach is used. In particular, in each of the other images30, a search region around the predicted position of the designatedtarget image point is selected. This search region may be defined simplyby using the position predicted by the homography as the centre point,with the width and height of the search region being proportional to thewidth and height of the image patch by a predetermined factor (e.g.twice). The homography is used to warp the image patch in the firstimage (typically 16×16 pixels) into the coordinate system of the secondimage.

Then, in respect of each search region, there is detected an image patchthat matches an image patch at the position of the designated targetimage point in the one of the images in which it is designated. Templatematching may be used, which involves sliding this warped image patchacross all pixel coordinates in the second image search region,computing a similarity score at each point, and detecting a match whenthe similarity score represents maximum similarity. A suitablesimilarity score is an L2-norm difference measure (which representsmaximum similarity by a minimum score). Another suitable similarityscore is a normalized cross-correlation/

Step S5-3 identifies the set of target image points are identified asbeing at the position of the designated target image point and atpositions of matching patches in the other images detected in step S5-3.The positions may be refined to sub-pixel resolutions as follows toavoid drift in the tracked target. The sub-pixel refinement is toapproximate the 1-neighbourhood of the difference cost minimum with aquadratic surface fit. If this surface yields a stable minimum, therefined sub-pixel position of the minimum as the tracked targetposition. If, however, the surface contains a saddle point, targettracking is stopped on the assumption that the target has becomeoccluded or otherwise disappeared from view.

Our other criterion for termination of target tracking is to check thelocation of the search patch. When it overlaps with the edge of theimage, tracking ceases.

In practice, step S5 may achieve a target track sustained accurately formany consecutive images 30.

In the modified method, steps S1 and S2 are performed as describedabove, and the bundle adjustment of step S4 is also performed in thesame manner, except that it is performed on the set of target imagepoints, i.e. the target track, together with the identified sets ofimage points derived from image features. As a result, the estimatedparameters represent the position of the target object point, togetherwith the positions of the object points. This means that the position ofthe target object point is derived intrinsically by the bundleadjustment, without the need to perform step S4.

In this modified method, error estimation may be performed as follows.

There is approximated the point error in the estimated position of thetarget image point arising from an assumption of Gaussian image noisewith standard deviation σ_(x), using the Schur complement of the Markovblanket of the target feature parameter. This gives us a 3D pointcovariance matrix for the target point with feature index t, inconditioned world coordinates, with the formula:

$\Lambda_{XYZ} = {\begin{bmatrix}a & b & c \\d & e & f \\g & h & i\end{bmatrix} = \left( {\frac{1}{\sigma_{x^{2}}}\left( {V_{tt} - {W_{t}^{T}U_{t}^{- 1}W_{t}}} \right)} \right)^{- 1}}$

with V_(tt) being the t'th diagonal 3×3 block of the full V featureinformation matrix, and U_(t), W_(t) being those portions of the full Uand W matrices that refer to cameras containing the target track (thatis, the feature with index t).

This error estimate (which is purely derived from image noise) may beextended with an assumption of some Gaussian-distributed systematicerror in the GPS signal. As we are only interested in the X and Ycomponents of error, the 2×2 upper left block of the 3D covariancematrix is extracted before adding the GPS covariance matrix andcomputing the standard deviation matrix via an SVD:

$\Lambda_{XY} = \begin{bmatrix}a & b \\d & e\end{bmatrix}$ Λ_(total) = Λ_(XY) + Λ_(gps) = U Σ V^(T)$\Sigma = \begin{bmatrix}\sigma_{1}^{2} & 0 \\0 & \sigma_{2}^{2}\end{bmatrix}$ $\sigma_{total} = {{U\begin{bmatrix}\sigma_{1} & 0 \\0 & \sigma_{2}\end{bmatrix}}V^{T}}$

This error estimate may be visualised by drawing a 3σ ellipse around thetarget location on the ground.

Experimental trials of the system have been performed over various ruraland semi-urban scenes 2. Utilising the modified method of FIG. 7 toanalyse the captured images 30 targets. This produced estimatedpositions of the target object which were generally in the range of 3-6m of the actual position of the target object as surveyed by a GPSposition measurement at the actual target, with 3σ error ellipse majorradii generally in the range of 6-9 m. This demonstrates accuratedetermination of position.

Furthermore, the run time of the algorithm was measured, using an IntelCore i7 870 CPU as the processor of the analysis apparatus, with asingle core applied to both the feature tracking of step S2 and thebundle adjustment of step S3. Typical run times for the feature trackingwere of the order of 250 ms per image 30. Typical run times for thebundle adjustment were 1 s, 6.1 s, 9.1 s and 16.8 s for a bundleadjustment where the number of images 30 was 40, 80, 120 and 160,respectively. Using 160 images 30, typically the entire method wasperformed in about 24 s. Thus, for many applications, and in particularin comparison with the image capture times it can be seen that themethod is effectively performed in real time.

REFERENCES

References [1] to [9] cited above are identified as follows and areincorporated herein by reference:

-   [1] Motion Imagery Standards Board: UAS datalink local metadata set,    Standard 0601.4, 4 Mar. 2010.-   [2] Motion Imagery Standards Board. Photogrammetry metadata set for    digital motion imagery, Engineering Guideline (EG) 0801.2, December    2009.-   [3] Jianbo Shi and Carlo Tomasi, “Good features to track”, In 1994    IEEE Conference on Computer Vision and Pattern Recognition    (CVPR'94), pages 593-600, 1994.-   [4] Krystian Mikolajczyk and Cordelia Schmid, “A performance    evaluation of local descriptors”, IEEE Trans. Pattern Anal. Mach.    Intell., 27(10):1615-1630, October 2005.-   [5] Muja and Lowe, “Fast approximate nearest neighbors with    automatic algorithm configuration”, International Conference on    Computer Vision Theory and Application VISSAPP'09, pages 331-340.    INSTICC Press, 2009.-   [6] Martin A. Fischler and Robert C. Bolles, “Random sample    consensus: a paradigm for model fitting with applications to image    analysis and automated cartography”, Commun. ACM, 24(6):381-395,    June 1981.-   [7] Levenberg, “A Method for the Solution of Certain Non-linear    Problems in Least-Squares”, Quarterly of Applied Mathematics, Vol.    2, No. 2, July 1944, 164-168.-   [8] Marquardt, “An Algorithm for the Least-Squares Estimation of    Nonlinear Parameters”, SIAM Journal of Applied Mathematics, Vol. 11,    No. 2, June 1963, pp. 431-441.-   [9] S Marandi and V Modi, A preferred coordinate system and the    associated orientation representation in attitude dynamics, Acta    Astronautica, 15(11):833-843, 1987.-   [10] Changchang Wu, Sameer Agarwal, Brian Curless, and Steven M    Seit, “Multi-core bundle adjustment”, Computer Vision and Pattern    Recognition (CVPR) 2011, 10(x):3057-3064, 2011.-   [11] H. B. Nielsen, “Damping parameter in Marquardt's method”,    Technical report, Informatics and Mathematical Modelling, Technical    University of Denmark, DTU, Richard Petersens Plads, Building 321,    DK-2800 Kgs. Lyngby, April 1999.

1. A method of determining the absolute position of at least one targetobject point in a scene, the method using a series of images of thescene with overlapping fields of view captured by a camera in positionsarranged in at least two dimensions across the scene and position datarepresenting the absolute positions of the camera on capture of therespective images, the method comprising: analysing the images toidentify sets of image points in plural images of the series of imagescorresponding to common object points in the scene; and performing abundle adjustment on the identified sets of image points that estimatesparameters representing the positions of the object points relative tothe positions of the camera on capture of the respective images takinginto account the absolute positions of the camera represented by theposition data as a prior without deriving modified estimates of thepositions of the camera, and without using input orientation datameasured by a sensor associated with the camera representing theorientation of the camera on capture of the images to set initialestimates of the orientations of the camera on capture of the respectiveimages, and deriving the absolute position of the target object point onthe basis of the results of the bundle adjustment referenced by theabsolute positions of the camera on capture of the respective imagesrepresented by the position data.
 2. A method according to claim 1,wherein the scene is on the ground and the series of images are imagescaptured from a camera on an aircraft.
 3. A method according to claim 1or 2, wherein the position data is derived from a global positioningsatellite receiver.
 4. A method according to any one of the precedingclaims, further comprising receiving user input designating at least onetarget image point in at least one of the images, that corresponds tothe at least one target object point.
 5. A method according to claim 4,wherein the step of deriving the absolute position of the target objectpoint comprises interpolating the position of the target object pointfrom the estimated positions of object points corresponding to a groupof image points located around the designated target image point.
 6. Amethod according to claim 4, wherein the method further comprisesidentifying a set of target image points in plural images of the seriesof images corresponding to the designated target image point and henceto the target object point; and the bundle adjustment is performed onthe set of target image points, together with the identified sets ofimage points, and the estimated parameters represent the position of thetarget object point, together with the positions of the object points,relative to the positions of the camera in respect of each image,whereby the position of the target object point is derived intrinsicallyby the bundle adjustment.
 7. A method according to claim 6, wherein thestep of identifying a set of target image points in plural images of theseries of images comprises: estimating a homography between said one ofthe images in which the designated target image point is designated andother images; predicting the position of the designated target imagepoint in the other images using the homography; detecting matching imagepatches within a search region around the predicted position of thedesignated target image point in the other images that match an imagepatch at the position of the designated target image point in said oneof the images, the set of target image points being identified as beingat the position of the designated target image point and at positions ofmatching patches in the other images.
 8. A method according to any oneof the preceding claims, wherein the step of analysing the images toidentify the sets of image points comprises: detecting features withinthe images; generating descriptors in respect of each image fromrespective patches of the image at the position of each detectedfeature; and detecting sets of corresponding descriptors generated fromdifferent images, the positions of the detected features from which aset of corresponding descriptors are generated being identified as a setof image points corresponding to a common object point.
 9. A methodaccording to claim 8, wherein the detected features are corner features.10. A method according to claim 8 or 9, wherein the features aredetected at plural scales.
 11. A method according to any one of claims 8to 10, wherein the step of detecting corresponding descriptors generatedfrom different images comprises detecting sets of descriptors generatedfrom different images that match each other and optionally that alsoconform to a common mapping between the positions in the respectiveimages of the features corresponding to those descriptors
 12. A methodaccording to any one of the preceding claims, wherein the bundleadjustment also estimates parameters representing the orientations ofthe camera in respect of each image.
 13. A method according to any oneof the preceding claims, wherein the bundle adjustment takes intoaccount the absolute positions of the camera in respect of each imagerepresented by the position data as a prior.
 14. A method according toany one of the preceding claims, wherein the bundle adjustment does notestimate parameters representing the positions of the camera in respectof each image.
 15. A method according to any one of the precedingclaims, wherein the bundle adjustment uses stored data representing theintrinsic properties of the camera.
 16. A method according to any one ofclaims 1 to 14, wherein the bundle adjustment estimates parametersrepresenting the intrinsic properties of the camera.
 17. A methodaccording to any one of the preceding claims, wherein the bundleadjustment is performed with initial estimates of the orientations ofthe camera on capture of the respective images that are within 90° ofeach other.
 18. A method according to any one of claims 1 to 16, whereinthe bundle adjustment is performed with initial estimates of theorientations of the camera on capture of the respective images thatassume each image is captured with the optical axis aligned with acommon object point in the scene.
 19. A method according to any one ofthe preceding claims, wherein the positions are arranged along a curvedpath.
 20. A method according to claim 19, wherein the bundle adjustmentis performed with initial estimates of the positions of the objectpoints relative to the positions of the camera in respect of each imagethat are derived from a geometrical measure of the curvature of the pathof the absolute positions of the camera in respect of each image.
 21. Amethod according to any one of the preceding claims, further comprisingcapturing the images and acquiring the position data.
 22. An apparatusfor determining the absolute position of at least one target objectpoint in a scene, the apparatus being configured to receive a series ofimages of the scene with overlapping fields of view captured by a camerain positions arranged in at least two dimensions across the scene andposition data representing the absolute positions of the camera oncapture of the respective images, wherein the apparatus is configured toanalyse the images to identify sets of image points in plural images ofthe series of images corresponding to common object points in the scene;and the apparatus is further configured to perform a bundle adjustmenton the identified sets of image points that estimates parametersrepresenting the positions of the object points relative to thepositions of the camera on capture of the respective images taking intoaccount the absolute positions of the camera represented by the positiondata as a prior without deriving modified estimates of the positions ofthe camera, and without using input orientation data measured by asensor associated with the camera representing the orientation of thecamera on capture of the images to set initial estimates of theorientations of the camera on capture of the respective images, theapparatus being configured to derive the absolute position of the targetobject point on the basis of the results of the bundle adjustmentreferenced by the absolute positions of the camera on capture of therespective images represented by the position data.
 23. A system fordetermining the absolute position of at least one target object point ina scene, the system comprising an apparatus according to claim 22 and anaircraft comprising a camera configured to capture said series of imagesand a position sensor configured to acquire said position data, theapparatus and aircraft comprising datalink interfaces for communicatingthe captured series of images and position data over a datalink.